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(54) Speaker authentication method and system from speech models 



(57) Speaker authentication method in a voice por- 
tal (3), comprising the following successive steps: 

(1 ) storing at least an utterance of one user of said 
portal in said portal (22), 

(2) subsequently requesting said user to utter choic- 
es in a voice menu (26), 



(3) subsequently adapting said user's speech mod- 
els with said choices uttered by said user (50), 

(4) using said adapted speech models for authenti- 
cating said user using said utterance stored in said 
portal (48). 
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Descriptl n 

[0001] Th pres nt invention concerns a speaker authentication m th d and syst m. More specifically, the present 
inv ntion concerns a multi-pass speech recognition and sp ak r authentication method for us r auth ntication in a 
5 voice portal, and an improved voice portal. 

Prior art 

[0002] Voice portals, for example in interactive voice response systems, are already known in which information can 
10 be stored and accessed by a plurality of users with spoken requests directly uttered in a microphone or transmitted 
through the telephone network and interpreted in the voice portal. Voice portals have the advantage that the information 
can be accessed without any prior knowledge and without any special equipment from any telephone terminal in the 
world. 

[0003] Voice portals often can give access to confidential information, for example bank or medical information. The 
15 level of confidentiality, and the kind of information which may be stored in a voice portal, thus critically depend on a 
reliable identification of the authorized users, in order to authorize them or not 

[0004] In many conventional systems, users have to enter or to confirm their identity with a user identification and/ 
or a password entered as DTMF codes on the keypad of the users' equipment. In many situations, like e.g. in cars, 
this solution proves unpractical. Moreover, user identifications and passwords are hard to remember and are often 

20 written down near the user's equipment, resulting in weak security. 

[0005] It has already been suggested to authenticate users with a spoken identity uttered at the entry point of the 
dialogue with the voice portal. Speech recognition techniques have been used for recognizing the claimed identity and 
speaker verification techniques for verifying the claimed identity. In the next dialogue turn, the user is accepted or 
rejected if he is not registered in the portal or if the claimed identity cannot be verified. 

2s [0006] Currently, the major trend in speech recognition and speaker verification. technology is the Hidden Markov 
Model. This statistical method describes speech by a sequence of states (Markov states). These are linked by transition 
probabilities, to move from state to state and each state emits, via a certain probability distribution, a speech vector. 
Among the many weaknesses of HMMs, one could mention the poor robustness against a typical mismatch situation 
(the training and testing speech material is different). In the last years of research in the field of speech recognition, 

30 many researchers have concluded that the robustness of current methods is far from being comparable to what human 
beings could achieve. 

[0007] Different methods have already been described to improve the robustness of speech recognition. It is known 
that the speaker verification accuracy improves when more speech material is available. Furthermore, some systems 
achieve better performance by using speaker-dependent speech models, which are usually learned during an enrol I - 
35 ment session. However, most users would not accept having to spend the time necessary for this session. Besides, 
speaker-dependent recognition can only be used once the speaker has been identified, but not for recognizing the 
identity claimed by the user during the first turn of a dialogue. 

[0008] Instead of learning a complete set of speech models for each user, several adaptation techniques have been 
described for adapting a good speaker-independent model set to a new speaker. If the adaptation data is available in 

40 one block, e.g. from a speaker enrollment session, one may use static adaptation methods. Alternatively, adaptation 
can proceed incrementally as adaptation data becomes available (incremental adaptation). In the case where the true 
transcription of the adaptation data is known, then it is termed supervised adaptation, whereas if the models are adapted 
with unlabelled speech material, then it is termed unsupervised adaptation. Known adaptation techniques comprise 
maximum likelihood linear regression (MLLR) techniques and maximum a posteriori (MAP) adaptation. 

45 [0009] Other adaptation techniques do exist to adapt the models with the current noise captured from the line. Many 
systems are using a short moment of silence to do the modeling. That particular model is then combined with the 
original one. A two-pass system is described in the patent application EP-A2-08801 26 which adapts the silence models 
by first segmenting the speech utterances and then adapting the silence modeling via unsupervised adaptation. 
[0010] In a voice portal, all those adaptation techniques may be used for improving the speech models during the 

so enrollment session and/or during the whole dialogue; the adapted models may even be stored in the system and used 
for speech recognition in future sessions. However, they cannot be used in each session until the speaker's identity 
has been recognized and/or verified. 

[0011] This first dialogue turn is thus very critical since only speaker-independent recognition techniques can be 
used and since the perplexity (number of possible branching) of the identity input is, at least, proportional to the number 
55 of users. Conversely, in the rest of the spoken dialogue, the speech recognition has an easier task; the perplexity is 
lower. 
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Summary of th invention 

[0012] On aim f th invention is to provid av ice portal syst m and method with improved speak rauth ntication 
p rformances over xisting methods and systems. 
5 [0013] On oth r aim is to provide a speak r-d pendant verification method m a voic portal syst m which allows 
for an accurate speaker recognition and verification based on an utterance spok n by th user during the first dialogue 
turn. 

[0014] In accordance with one embodiment of the present invention, a speaker authentication method is provided 
in which a special dialogue has been designed in order to collect more data and to adapt the speech and silence models 
10 used for the first dialogue turn. 

[0015] According to another aspect of the invention, a multi-pass speech recognition scheme is used to improve the 
adaptation of speech models. 

[0016] According to another aspect of the invention, a speaker authentication method in a voice portal is provided, 
which comprises the following successive steps: 

15 

(1) storing at least an utterance of one user of said portal in said portal, 

(2) subsequently requesting said user to utter choices in a voice menu, 

(3) subsequently adapting said user's speech models with said choices uttered by said user, 

(4) using said adapted speech models for authenticating said user using said utterance stored in said portal. 

20 

[001 7] According to another aspect of the invention, a spoken dialogue is constructed that enables a delayed answer 
upon the user's speech input. As a matter of fact, in most applications of voice portals, a reliable user authentication 
is only needed for the delivery of security-critical information or for specific transactions that are usually carried out 
only after a few selections in the voice menu. Therefore, the user authentication can be postponed till that later point 
25 in the dialogue. 

[0018] Moreover, user authentication may be purely suppressed if the critical dialogue portion is not reached. 
[001 9] According to another aspect of the invention, the speech models used for speaker authentication are adapted 
during the spoken dialogue. This allows to perform the speaker authentication task with new speaker-dependant and 
session-dependant speech models, and thus to improve the authentication accuracy. 
so [0020] According to another aspect of the invention, the first utterance is stored in the system in order to postpone 
its recognition and the verification of the user's identity until enough speech material has been collected to perform a 
speaker-dependant speech recognition and speaker verification. The first utterance can be normalized, multi-pass 
adaptation techniques can be used on it and the speech models can be adapted in order to improve the speaker 
authentication and speech recognition quality. 

35 

Brief description of the drawing 

[0021] The invention will be described with reference to the following drawing wherein: 
40 Fig. 1 shows a schematized representation of a possible embodiment of a voice portal according to the invention. 

Fig. 2 shows a flowchart illustrating a dialogue with the inventive voice portal. 
Detailed Description 

45 

[0022] Fig. 1 shows a diagram of a telecommunication system including a voice portal hosting system 3. A plurality 
of users can establish a voice connection with this voice portal 3 over a.public telecommunication network 2, such as 
a public switched telecommunication network (PSTN), an integrated services data network (ISDN), a mobile telecom- 
munication network, e.g. a GSM network, or a voice-enabled IP-Network (VoIP). The connection is established by 

so selecting the voice porta) 3 phone number on a user terminal equipment 1 such as a phone terminal. Preferably, the 
voice portal phone number is a free business number in the 0800 range or, in a various embodiment, a charging number 
in the 0900 range. The voice portal 3 may give access to information and services from one or several service providers 
1 3 connected permanently or occasionally to the porta) over the second telecommunication network 1 2. 
[0023] The voice porta) hosting system includes a dialogue manager 11 for establishing a spoken dialogue with the 

55 user 1 . The dialogue manager 11 answers to commands or requests uttered by the user with the requested information 
or service or by establishing a new connection with an external server. 

[0024] User voic r qu sts ar recognized by a speech recogniz r 10 that pref rabty uses HMM (Hidden Markov 
Mod Is), adaptiv neural networks or hybrids n tworks. In a pr ferred embodim nt, the speech recognition modul 
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comprises Hidden Markov Models built with a toolkit such as th HTK toolkit. 

[0025] A s lector 9 can select speaker-indep ndent speech mod Is 7 or, as so n as the us r has been id ntified, 
us r-depend nt speech mod Is 8. User-dep ndent speech models usually allow for a fast randmor reliable speech 
recognition than user-indep ndent speech mod Is, but can bviously only b us d aft r us r identification. 

5 [0026] At least th first utterance sp k n by the us r during a sessi n, but preferably all th us r utterances, ar 
stored in a temporary memory 5 and us d by a speech mod Is adaptation module 60 for improving th quality of th 
speech recognition and speaker authentication performed by the modules 10 and 6 respectively. As will b explain d 
later, at least some utterances are stored in order to be recognized or to serve for an authentication using improvements 
of the speech models gained from subsequent utterances. 

10 [0027] The user is identified and preferably authenticated by a speaker authentication module 6. The speaker au- 
thentication module preferably uses HMM (Hidden Markov Models), but could possibly use adaptive neural networks 
or hybrids networks. In a preferred embodiment, the speaker authentication module 6 comprises Hidden Markov Models 
built with a toolkit such as the HTK toolkit. In a preferred embodiment, the user identity is first recognized from the text 
corresponding to the utterances spoken using speaker-independent speech models, for example using the speech 

15 models 7. An active verification signal Verif is delivered as soon as the identity of the calling user is known. Speaker- 
dependant speech models 8 can be selected by the selector 9 and used by the speech recognizer 10 as soon as the 
signal Verif is active. The user identity is then verified in order to authenticate the user with an improved reliability; an 
authentication signal Auth becomes active when the user has been authenticated with the required confidence level. 
[0028] Instead of collecting speaker-dependant speech models in the speaker-dependent database 8 during an en- 

20 rolment session, adaptation techniques can be applied for improving the models available for speech recognition and 
speaker authentication. In this case, by using only a small amount of data from a new user, the speaker-independent 
model 7 can be adapted using maximum likelihood linear regression (MLLR) to fit the characteristics of each user. This 
adaptation can proceed incrementally as adaptation data becomes available (incremental adaptation). 
[0029] The speech models used on each utterance for recognition and/or authentication are adapted during each 

25 session with the voice material stored in the temporary memory 5, as will be explained later. A second pass may be 
used on the first utterances for improving the recognition and/or authentication reliability using subsequently spoken 
voice material. The segmentation of the utterances may be improved with subsequent voice choices. A normalization 
of the speech segments may also be performed. New speech models, including for example new silence models, which 
are better adapted to the channel currently used, may be created from subsequent utterances and added to the set of 

30 models used for recognizing the first utterances and for authenticating the user from those utterances. 

[0030] The speaker authentication module 6 and the speech recognizer 1 0 can be combined into one single module. 
The speaker authentication module may use additional information, for example data transmitted over a signalization 
channel such as the caller terminal identification, for example the CLI in afixed network, the IP address in an Internet 
network or the IMSl (International Mobile Subscriber Identification) in a mobile network, in order to make easier the 

35 user identification/authentication task. 

[0031] The confidence level needed for authenticating the user can preferably be set by the dialogue manager and/ 
or by the operator of the voice portal 3, for example a service provider 13, depending on the requested security. A 
USER signal unambiguously indicates the user identified and/or authenticated by the module 6. 
[0032] Several methods can be used independently or simultaneously for improving the authentication and recog- 

40 nition of the first utterance based on subsequent utterances: 

1 . Subsequent normalization techniques 

[0033] Several normalization techniques (CMN, Energy normalization) can be applied within the inventive dialogue 
45 design. These normalization techniques work only when enough speech material is available to estimate the needed 
values of energy means or cepstral coefficients. In conventional voice response systems, these techniques are not 
used because they induce unacceptable delays in the dialogue interaction. 

[0034] According to the invention, the user's first utterance is normalized using subsequent speech material, for 
example choices in a voice menu, uttered by the user. This allows for a more accurate speech recognition and speaker 
so identification/verification algorithm. 

2. Multi-pass adaptation techniques 

[0035] Figure 2 illustrates a specifically adapted dialogue between the user 1 and the inventive voice portal 3. The 
55 dialogue has been adapted in the sense that the dialogue manager will activate the final pass of the speech recognition 
and authentication process only when the user reaches a critical part of the dialogue. 

[0036] The one skilled in the art will understand that, besides th improved us r-lnt rface.wh n this dial gu design 
is applied, many fficient speech recognition techniques can be appli d to improv th perceived quality f the voice 
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interaction. 

1 . During th first step 20, at th ntry p int of a p rsonal voice portal, th us r selects th voice portal 3 phone 
number in the network 2 and th call is established. The user is then requested to utter his us r identification and/ 
5 or password. The speech mat rial and chann I n is is collected and st r d in th temporary memory 5 during 

step 22. No decision about th rejection of the user is made at this p int of th dialogue. Th claimed identity will 
only be verified later on, with known speak rv rification meth ds.whenth f llowing conditions have be nm t: 

■ The user wants to access a secure portion of the voice menu, and 

10 

■ The user's speech models have been adapted with subsequent speech material, and/or 

■ Enough speech material has been collected from said user to allow for reliable speaker verification. 

is 2. Two processes are then performed in parallel in the voice portal 3. In the first process, the user Is given access 

to a free portion of the voice menu (step 24). This free portion may include for example access to non-confidential 
information from the service provider 13. The user can utter choices in this menu (step 26); speech material is 
collected in the memory 5. The dialogue proceeds with subsequent questions and answers by using speaker- 
independent speech recognition (step 30) as long as the user has not been identified. 

20 in a preferred embodiment, the user is provisionally recognized during steps 26^30 using for example a speak- 

er-independent recognition of the text corresponding to the first utterance or non-speech elements such as the 
CLI, IMSI or IP terminal identification. The provisory identity is used in order to retrieve from the user profile data- 
base the user's speech andtor language models 8 used for improving the speech recognition accuracy during the 
dialogue, and will be verified later on, when needed, using known speaker verification methods in order to authen- 

25 ticate the user. Test 28 denotes a check of the Verif signal, which indicates if the user has already been identified 

and if speaker-dependant recognition algorithms can be used during step 32 for speech recognition. 

In another embodiment, an incremental adaptation technique is used for improving the speech model used 
for each turn of the dialogue. Supervised adaptation techniques are preferably used if the user's answers can be 
expected (for example for choices in a closed menu list). 

30 

3. In a concurrent task within the system, the authentication flag Auth is initially reset (step 42). During step 44, 
the authentication module then tries to identify (step 440) and to authenticate (step 442) the user 1 using the first 
utterance spoken during the first exchange. Apart from the user's voice, other elements such as the terminal iden- 
tification (CLI, IMSI, IP-address, etc.) may be used in order to help in the identification and authentication process. 

35 At least the first utterance, but preferably all speech inputs, is first segmented via a speaker-independent 

speech recognizer. The speech segments and non-speech signals are then used to adapt the speech/silence 
models via an adaptation technique like for instance MLLR. 

4. For speech models, only the segments with an acceptable confidence level are used to provide supervised 
40 adaptation of model parameters. Unsupervised techniques may also be applied. 

5. Subsequent recognition/adaptation passes may be performed during step 50 to further improve (0 the confidence 
level of the recognition and authentication system, (ii) the segmentation, and (Hi) the adaptation of speech and 
silence models. In the case where the user number is composed of a digit string, the subsequent recognition 

45 passes are stopped when the error detection code grants permission. 

6. As the dialogue reaches the critical security point (step 34) in the menu where a user authentication is requested 
in order for example to access more confidential information or to perform financial transactions, the final recog- 
nition and verification pass is processed and the user may be rejected from the service at this point 

so |f the user has already been authenticated at this stage, he is given access to the secure portion of the voice 

menu (step 40). Otherwise, access is denied, or the user may be given a second try or requested to authenticate 
himself using another method, for example by uttering a password or entering a DTMF-PIN code on the keypad 
of his equipment (step 38). 

In the preferred embodiment illustrated in Figure 2, a succession of authentication passes 44 are iteratively 
55 performed while the user is navigating through the voice menu, until a predefined confidence level has been 

reached (test 46). The authentication flag is set when this level has been reached (step 48); otherwise an additional 
adaptation pass is p rform d (st p 50). No access to secure p rtions of this m nu is all wed until th user has 
b nr liably authenticat d. It is also possible to build a voice menu with a plurality of portions with different security 
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r quirem nts, different confid nee lev Is for the user authenticatl n being required for accessing said different 
portions. 

Thus, according to the invention, a v ry reliabl user v rification and a v ry accurat recognition f th first 
utterance can be performed, based on more sp ech material and using speech models adapted t th current 
sp aker. 

7. in the dialogue, there is no turn to explicitly accept the user. He is accepted by default. 

The method uses the well-known fact that the accuracy of speaker-dependent speech recognition is higher 
than that of speaker-independent systems. 

In the case of personal voice portals, the present invention uses several spoken utterances (as many as 
possible) to adapt the speech models to the speaker, and therefore postpones the decision till the user wants to 
access the secure portion of the voice menu. In a sense, it acts precisely as a fast on-line enrollment session. 

The techniques 1 (subsequent normalization) and 2 (multi-pass adaptation) may be combined to achieve 
increased performance. 

3. Improved speaker verification 

[0037] During the elapsed time between the first dialogue turn and the critical security point, several command words 
may be used to complement the rejection/acceptation decision of the server system. In order to implement this addi- 
tional verification scheme, the spoken command words of each session have to be stored within the authentication 
system and the authentication system extracts from them the model needed for the speaker verification procedure. In 
fact, these command words are used to create additional speech material for speaker verification. It is well known that, 
just as the speaker adaptation improves with the availability of more data, speaker verification improves equally with 
the increasing quantity of speech material. 

[0038] The combination of all three methods contributes to improve the overall perceived quality of service. 
[0039] As mentioned above and from the dialogue perspective, another obvious advantage is to improve the dialogue, 
since the rejection/acceptation dialogue is simply skipped in most cases. The dialogue is shorter and therefore more 
efficient. 

[0040] In some cases and for some particular model parts, a higher adaptation speed has been measured with 
simpler models. The technique is used, for instance, for silence models adaptation. 

Example of implementation 



[0041 ] At least some of the voice applications pen* ormed by the dialogue manager include voice-enabled e-commerce 
applications. The following example of implementation is about a banking teleservice. The user wishes to know the 
amount of money left on his bank account. 



The user 


The banking teleservice 


The user calls the banking service via the voice portal 
and from his hotel room. 






The system: Welcome to the banking service, please 
state your user number. 


The user: 346 1 2493 






The system: What kind of service do you want 
Information on new services, information about your 
account, or would you like to transfer money? 


The user: / would like information about my account 






The system: Would you like to know the current account 
status or the latest transactions that have occurred. 


The user. / would like to know the current account status 
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(continu d) 



The user 


The banking teleservice 




Th v rification fth claimed id ntityisnowp rf rm d. 
Th three sentences ar used to adapt th sp ech and 
silence mod Is. Additionally, the auth nttaation syst m 
uses the digit string of the user number as well as the 
two subsequent sentences to infer if the user's claimed 
identity is correct. 




The system; On the account number 346 1 2493, the 
available amount is 4345.80 CHE 


The user terminates the communication. 
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[0042] The voice portal 3 may comprise a standard digital computer including an internal memory (not shown) in 
which a computer program product can be directly loaded for performing the method steps of the invention when said 
program is run on said computer 

Claims 

1 . Speaker authentication method in a voice portal (3), comprising the following successive steps: 

(1) storing at least an utterance of one user of said portal in said portal (22), 

(2) subsequently requesting said user to utter choices in a voice menu (26), 

(3) subsequently adapting said user's speech models with said choices uttered by said user (50), 

(4) using said adapted speech models for authenticating said user using said utterance stored in said portal 
(442). 

2. Method according to claim 1 , further repeating said steps 2 to 4 until a predefined confidence level has been 
reached for the authentication of said user 

3. Method according to one of the preceding claims, in which the identity of said user is first recognized and then 
verified using said adapted speech models for authenticating said user. 

4. Method according to one? of the preceding claims, in which the identity of said user is recognized from the text 
corresponding to said utterance. 

5. Method according to the preceding claim , in which the identity of said user is recogn ized from the text corresponding 
to said utterance using speaker-independent voice recognition algorithms. 

6. Method according to the claim 4, in which the identity of said user is recognized from the text corresponding to 
said utterance using said adapted speech models. 

7. Method according to one of the preceding claims, in which the identity of said user is recognized using text-inde- 
pendent speaker verification. 

8. Method according to claim 3, in which the identity of said user is determined using data transmitted over the 
^ signaling channel. 

9. Method according to the preceding claim, wherein said data include the CLI terminal's identification. 

10. Method according to one of the preceding claims, wherein user-dependant speech models (8) are stored in said 
^ portal and retrieved as soon as said user has been identified in order to improve subsequent speech recognition 

and/or the verification of the user's identity. 

11. Method according to the preceding claim, in which said us r-dependant sp ech mod Is ar adapted using said 
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choices utt red by said us r. 

12. M thod according to on of the preceding claims, wh r inth identity of said user is v rifi d using said adapted 
speech models. 

5 

13. Method according to the prec ding claim, wh rein said user is auth nticated using supervised speech models 
adaptation algorithms. 

14. Method according to one of the preceding claims, wherein said user is authenticated and/or verified using Hidden 
10 Markov Models and wherein said speech models are adapted using an adaptation technique of Hidden Markov 

Models. 

15. Method according to the preceding claim, wherein said speech models are adapted using MLLR adaptation tech- 
niques. 

15 

16. Method according to the preceding claim, wherein said choices are used for adapting the speech models and 
therefore improving the segmentation of said utterance stored in said portal. 

17. Method according to one of the preceding claims, wherein said user's utterance is normalized with subsequent 
20 speech material from said user. 

1 8. Method according to one of the preceding claims, wherein new speech models are created from said subsequently 
uttered user choices. 

25 19. Method according to the preceding claim, wherein said new speech models indude silence models. 

20. Method according to one of the preceding claims, wherein said user is prompted to utter his identity immediately 
after a connection has been established with said voice portal, said identity being stored as said utterance. 

30 21 . Method according to one of the preceding claims, wherein said user is only authenticated when he wants to access 
a secured portion of said voice menu. 

22. Method according to one of the preceding claims, in which a plurality of portions with different security requirements 
have been defined in said voice menu, 

35 and in which different confidence levels for said user authentication are defined for accessing said different 

portions of said voice menu. 

23. Speech recognition method in a voice portal, comprising the following successive steps: 

40 (1 ) storing at least an utterance of one user of said system in said portal, 

(2) subsequently requesting said user to utter choices in a voice menu, 

(3) subsequently adapting the user's speech models with said choices uttered by said user, 

(4) subsequently using said adapted speech models for recognizing said utterance stored in said portal. 

45 24. Computer program product directly loadable into the internal memory of a digital computer, comprising software 
code portions for performing the steps of one of the preceding claims when said product is run on an interactive 
voice response system connected to a first telecommunication network. 

25. Voice portal (3), comprising: 

so 

a speaker authentication module (6) comprising Hidden Markov Models for authenticating users of said portal, 
a dialogue manager (11 ) for managing a dialogue between said system and users (1 ) of said system, wherein 
said dialogue enables access to at least one secured portion in said system, 
an adaptation module (60) for adapting said Hidden Markov Models, 

55 

wherein said dialogue manager enables a dialogue in which said user authentication only happens when 
said us r requests access to said secured portion, based on at I ast on first utt ranee uttered at th beginning 
of said dialogue and stored in said portal. 
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26. Voice portal according to claim 25, wherein said authentication is perform d with said at least one first utt ranee 
and using adaptation of sp ech models bas d n subsequ nt sp ech mat rial from said us r. 

27. Voice portal according to claim 25, wherein said at least one first utterance is normalized with subsequent speech 
5 material from said user. 

28. Voice portal according to claim 25, wherein said speech materia) is used for adapting the speech models and 
therefore improving the segmentation of said utterance stored in said portal. 

10 
15 
20 
25 
30 
35 
40 
45 
50 
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